Electro-optic vision systems

ABSTRACT

The present invention involves a vision system including devices and methods of augmented reality wherein an image of some real scene is altered by a computer processor to include information from a data base having stored information of that scene in a storage location that is identifed by the real time position and attitude of the vision system.

CROSS-REFERENCES TO RELATED PATENT APPLICATIONS

[0001] This application is a continuation-in-part of the followingpatent applications: Ser. No. 08/691,784, filed Aug. 2, 1996; Ser. No.08/119,360, filed on Sep. 10, 1993; Ser. No. 08/307,360, filed on Sep.14, 1994; Ser. No. 08/335,912, filed on Nov. 8, 1994, Ser. No.08/335,940, filed on Nov. 8, 1994; Ser. No. 08/335,710, filed on Dec. 4,1994; Ser. No. 08/441,299, filed on Mar. 27, 1995; Ser. No. 08/480, 689,filed on Jun. 7, 1995; Ser. No. 08/482,943, filed on Jun. 7, 1995; andSer. No. 08/571,096, filed on Dec. 12, 1995.

FIELD OF THE INVENTION

[0002] The present invention is generally related to electronic visiondevices and methods, and is specifically related to image augmentationin combination with navigation, position, and attitude devices.

BACKGROUND OF THE INVENTION

[0003] One may have to look quite far into the annals of history to findthe first uses of maps. Maps generally provide information to alert auser to things that are not readily apparent from simple viewing of areal scene from the users location. For example, a user of a city roadmap may not be able to see a tunnel on Elm Street if the user iscurrently seven miles away on First Street and looking in the directionof the Elm Street tunnel. However, from the First Street location, theuser could determine from a road map that there is a tunnel on ElmStreet. He could learn that the tunnel is three miles long, starts onEighth Street and ends on Eleventh Street. There may even be anindication of the size of the tunnel such that it could accommodate fourtraffic lanes and a bicycle lane.

[0004] Unfortunately, it is not always possible to translate theinformation from a map to the real scene that the information representsas the scene is actually viewed. It is common for users of maps toattempt to align the map to reality to get a better “feel” of wherethings are in relation to the real world. Those who are familiar withmaps can verify that the fact that maps are drawn with north beinggenerally in the direction of the top of the map, is of little use whentranslating the information to the scene of interest. Regardless ofwhere north is, one tends to turn the map so that the direction ahead ofthe user, or in the direction of travel, in a real scene matches thatdirection on the map. This may result in the condition of an “upsidedown” map that is quite difficult to read (the case when the user istraveling south). Although translating the directions of the map toreality is a formidable task, it is an even greater problem to translatethe symbols on the map to those objects in reality which they represent.The tunnel symbol on the map does not show what the real tunnel actuallylooks like. The fact that the appearance of the tunnel from infinitelymany points of view is prohibitively difficult to represent on a mapaccounts for the use of a simple symbol. Furthermore, the map does nothave any indication from which point of view the user will first see thetunnel, nor any indication of the path which the user will take toapproach the tunnel.

[0005] It is now possible to computerize city road map information anddisplay the maps according to the path taken by a user. The map isupdated in “real-time” according to the progress of the user through thecity streets. It is therefore possible to relieve the problem ofupside-down maps as the computer could re-draw the map with the text incorrect orientation relative to the user even when one is traveling in asoutherly direction. The computer generated map is displayed at amonitor that can be easily refreshed with new information as the userprogresses along his journey. Maps of this type for automobiles are wellknown in the art. Even very sophisticated maps with computer generatedindicia to assist the user in decision making are available anddescribed in patents such as DeJong U.S. Patent No. 5,115,398. Thisdevice can display a local scene as it may appear and superimpose ontothe scene, symbolic information that suggests an action to be taken bythe user. Even in these advanced systems, a high level of translation isrequired of the user. The computer generated map does not attempt topresent an accurate alignment of displayed images to the real objectwhich they represent.

[0006] Devices employing image supplementation are known and includeHead Up Displays (HUDs) and Helmet Mounted Displays (HMDs). A HUD is auseful vision system which allows a user to view a real scene, usuallythrough an optical image combiner such as a holographic mirror or adichroic beamsplitter, and have superimposed thereon, navigationalinformation for example symbols of real or imaginary objects, vehiclespeed and altitude data, et cetera. It is a primary goal of the HUD tomaximize the time that the user is looking into the scene of interest.For a fighter pilot, looking at a display device located nearby on aninstrument panel, and changing the focus of ones' eyes to read thatdevice, and to return to the scene of interest, requires a criticallylong time and could cause a fatal error. A HUD allows a fighter pilot tomaintain continuous concentration on a scene at optical infinity whilereading instruments that appear to the eye to also be located at opticalinfinity and thereby eliminating the need to refocus ones' eyes. A HUDallows a pilot to maintain a “head-up” position at all times. For theairline industry, HUDs have been used to land airplanes in lowvisibility conditions. HUDs are particularly useful in a landingsituation where the boundaries of a runway are obscured in the pilotsfield of view by fog but artificial boundaries can be projected onto theoptical combiner of the HUD system to show where in the user's visionfield the real runway boundaries are. The virtual runway projection ispositioned in the vision field according to data generated bycommunication between a computer with and the airport instrument landingsystem (ILS) which employs a VHF radio beam. The system provides thecomputer with two data figures. First a glide slope figure, and second,a localizer which is a lateral position figure. With these data, thecomputer is able to generate an optical image (photon) to be projectedand combined with the real scene (photon) that passes through thecombiner and thereby enhancing certain features of the real scene; forexample runway boundaries. The positioning of the overlay depends on theaccuracy of the airplane boresight being in alignment with the ILS beamand other physical limitations. The computer is not able to recognizeimages in the real scene and does not attempt to manipulate the realscene except for highlighting parts thereof. HUDs are particularlycharacterized in that they are an optical combination of two photonscenes. The combination being a first scene, one that is normally viewedby the users eyes passes through an optical combiner, and a second,computer generated photon image which is combined with the real image atan optical element. In a HUD device it is not possible for the computerto address objects of the real scene, for example to alter or deletethem. The system only adds enhancement to a feature of the real image bydrawing interesting features thereon. Finally, HUDs are very bulky andare typically mounted into an airplane or automobile and require a greatdeal of space and complex optics including holograms and speciallydesigned lenses.

[0007] HMDs are similar to HUDs in that they also combine enhancementimages with real scene photon images but they typically have veryportable components. Micro CRTs and small combiners make the entiresystem helmet mountable. It is a complicated matter to align computergenerated images to a real scene in relation to a fast moving helmet.HUDs can align the data generated image that is indexed to the slowmoving airplane axis which moves slowly in relation to a runway. Forthis reason, HMDs generally display data that does not change with thepilots head movements such as altitude and airspeed. HMDs suffer thesame limitation as the HUDs in that they do not provide the capacity toremove or augment elements of the real image.

[0008] Another related concept that has resulted in a rapidly developingfield of computer assisted vision systems is known as virtual reality(VR). Probably best embodied in the fictional television program “StarTrek; The Next Generation”, the “Holodeck” is a place where a user cango to have all of his surroundings generated by a computer so as toappear to the user to be another place or another place and time.

[0009] Virtual reality systems are useful in particular for a trainingmeans. For example in aircraft simulation devices. A student pilot canbe surrounded by a virtual “cockpit” which is essentially a computerinterface whereby the user “feels” the environment that may be presentin a real aircraft, in a very real way and perhaps enhanced withcomputer generated sounds, images and even mechanical stimuli. Actionstaken by the user may be interpreted by the computer and the computercan respond to those actions to control the stimuli that surround theuser. VR machines can create an entire visual scene and there is noeffort to superimpose a computer generated scene onto a real scene. A VRdevice generally does not have any communication between its actuallocation in reality and the stimuli being presented to the user. Thelocation of the VR machine and the location of the scene being generatedgenerally have no physical relationship.

[0010] VR systems can be used to visualize things that do not yet exist.For example, a home can be completely modeled with a computer so that apotential buyer can “walk-through” before it is even built. The buyercould enter the VR atmosphere and proceed through computer generatedimages and stimuli that accurately represent what a home would be likeonce it is built. In this way, one could know if a particular style ofhome is likable before the large cost of building the home is incurred.The VR machine being entirely programmed with information from adesigner does not anticipate things that presently exist and there is nocommunication between the elements presented in the VR system to thoseelements existing in reality.

[0011] While the systems and inventions of the prior art are designed toachieve particular goals, features, advantages, and objectives, some ofthose being no less than remarkable, these systems and inventions havelimitations and faults that prevent their use in ways that are onlypossible by way of the present invention. The prior art systems andinventions can not be used to realize the advantages and objectives ofthe present invention.

SUMMARY OF THE INVENTION

[0012] The present invention involves a vision system including devicesand methods of augmented reality wherein an image of some real scene isaltered by a computer processor to include information from a data basehaving stored information of that scene in a storage location that isidentified by the real time position and attitude of the vision system.

[0013] One embodiment of the present invention comprises an imageprocessing system used in an electro-optic apparatus having an imagecapturing means, position determining means, attitude determining means,database of real scene information, and a display means. The imagecapturing means generating a digital image representing an object. Theimage processing system comprises a data processor and a graphicprocessor generating an image to be displayed by the display means basedon data generated by the data processor. The image processing systemfurther comprises means for delivering the digital image to the dataprocessor and means for identifying information related to the digitalimage from the database information. The related information isdelivered to the data processor. The data processor combining andprocessing the digital image and the related information. As a result, auser of the electro-optic apparatus can see an augmented image

[0014] These and other features of the present invention will becomeapparent from the following description when read in conjunction withthe drawings and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015]FIG. 1 is a block diagram of an electro-optic system of thepresent invention.

[0016]FIG. 2 is a block diagram showing an image processing system ofthe present invention used in the electro-optic system of FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

[0017] The present invention comprises a novel electro-optic system andassociated image processing devices. The following description ispresented to enable any person skilled in the art to make and use theinvention. Descriptions of specific applications are provided only asexamples. Various modifications to the preferred embodiments will bereadily apparent to those skilled in the art, and the general principlesdefined herein may be applied to other embodiments and applicationswithout departing from the spirit and scope of the invention. Thus, thepresent invention is not intended to be limited to the embodimentsshown, but is to be accorded the widest scope consistent with theprinciples and features disclosed herein.

[0018]FIG. 1 is a block diagram of an electro-optic system 10 of thepresent invention. System 10 comprises a position determining means 16,an attitude determining means 15, a camera 19, an image processing unit14, and a display 13. Camera 19 comprises an electro-optic devicecapable of converting a photon input (from a field of view) into anelectronic image. Camera 19 then transmits the electronic image to imageprocessing unit 14. Position determining means 16 determines theposition of the camera and transmit the information to image processingunit 14. Similarly, attitude determining means determines the camerapointing attitude of the camera defined by, e.g., the symmetry axis ofthe camera field of view. Attitude determining means 15 transmits theattitude information to image processing unit 14. Image processing unit14 analyzes and processes inputs from position determining means 16,attitude determining means 15, and camera 19 so as to generate real andaugmented images. These images are then transmitted to display 13 foruse by the user. Various applications and advantages of electro-opticsystem 10 have been disclosed in the following copending patentapplications: Ser. No. 08/119,360, entitled “An Electro-Optic VisionSystem Which Exploits Position and Attitude” and filed Sep. 10, 1993;Ser. No. 08/307,360, entitled “Vision System For Viewing A SportingEvent” and filed Sep. 14, 1994; Ser. No. 08/335,912, entitled “VisionImaging Devices and Methods Exploiting Position And Attitude” and filedNov. 8, 1994; Ser. No. 08/335,940, entitled “Vision Imaging Devices AndMethods Having An Unlimited Zoom Range” and filed Nov. 8, 1994; Ser. No.08/335,710, entitled “Computer Games Having Optically Acquired ImagesWhich Are Combined With Computer Generated Graphics And Images” andfiled Dec. 4, 1994; Ser. No. 08/441,299, entitled “Augmented RealityVision Systems Which Derive Image Information From Other Vision Systems”and filed Mar. 27, 1995; Ser. No. 08/480,689, entitled “Vision SystemsFor Viewing Objects That Identify Themselves” and filed Jun. 7, 1995;Ser. No. 08/482,943, entitled “An Electro-Optic Vision System WhichExploits Position And Altitude” and filed Jun. 7, 1995; Ser. No.08/482,944, entitled “Vision System Computer Modeling Apparatus” andfiled Jun. 7, 1995; Ser. No. 08/571,096, entitled “Computer VisionSystem Which Determines Identify, Position and Orientation Of Objects InA Scene And Displays Augmented Images Thereof” and filed Dec. 12, 1995.These patent applications are incorporated herein by reference.

[0019] It should be appreciated that electro-optic system 10 of FIG. 1may contains additional components. For example, it may be desirable toinclude a set of user inputs so that an user can enter data to imageprocessing unit 14. Other measuring devices, such as temperature gauge,accelerometer, range finder, etc., can also be coupled to imageprocessing unit 14.

[0020]FIG. 2 is a block diagram of an embodiment of an electro-opticsystem 100 showing the detail structure of its image processing unit.System 100 comprises a video input device, such as a camera 102, whichaccepts optical image information and generates a corresponding digitalvideo in the form of IEEE 1394 (FireWire) data. Camera 102 canoptionally generate information relating to iris, exposure time, zoom(image magnification) ratio and stabilizer data (e.g., 1, 2, or 3 axisacceleration). FireWire is a high speed serial interface busspecification especially tuned for transferring digital videoinformation at rates of up to 400 Megabits per second (Mbits/second).This interface has been implemented in Sony Corporation's DigitalHandyCam series of camcorders. An exemplary camera contains one or morecharged coupled device (CCD) area imaging sensors. It producesinformation which is digitized and encoded to an industry standard videoformat. This data is then transmitted serially at up to 400 Mbits/secondvia the FireWire data bus. Control and configuration information can bepassed bi-directionally over this bus as well.

[0021] It should be noted that although IEEE 1394 is the presentlypreferred digital interface for video data, the invention could beimplemented using other digital interfaces, now available or to bedeveloped in the future.

[0022] The output of camera 102 is connected to a FireWire videointerface chipset 104. This chipset accepts the IEEE 1394 data fromcamera 102 and generates command and data in a Zoom Video bus 106 and aPeripheral Component Interconnect (PCI) bus 108. Zoom Video (or ZoomedVideo) is an interface standard used by Personal Computer ManufacturingCard Industry Association (PCMCIA) card manufacturers and graphicscontroller manufacturers to provide a central processing unit (CPU)independent path to a graphics controller for digital video information.It is a bit parallel (YUV encoded) serial data interface (i.e., thepixels arrive serially for each line). Vertical, horizontal sync as wellas dotclock is provided. PCI bus 108 is a 32 (expandable to 64) bit highspeed parallel databus standard. The bus operates currently at 33 Mhz soas to provide up to 132 Mbytes/second data transfer rate. Futureimplementations will increase the bus speed. This bus is described indetail in the “PCI Specification, version 2.1” published by the PCISpecial Interest Group. Chipset 104 receives the IEEE 1394 serial datastream (at up to 400 Mbits /second) and converts the data into paralleldata suitable for sending over the PCI and/or Zoom Video data busses.The data that goes out on Zoom Video bus 106 is arranged to fit the YUVencoding format with vertical and horizontal syncs. Chipset 104 can alsooperate as a PCI Bus Master, thus can burst image data to anywhere inthe CPU's main memory or to any PCI slave device's memory.

[0023] It should again be pointed out that Zoom Video bus 106 and PCIbus 108 are exemplary digital buses. The present invention could beimplemented using other high bandwidth buses.

[0024] The output of chipset 104 is coupled to an image processor (IP)110. The Zoom Video bus allows IP 110 to receive video data in parallelwith a CPU system, shown in FIG. 2 as reference numeral 114. IP 110could perform many tasks, from complex to simple. At the complex end, IP110 may be responsible for processing video on a frame by frame basis toextract data from, or to enhance the image. An example is the V-LACE™real time image enhancement algorithm from DigiVision. At the simpleend, IP 110 may be asked to shut down and passively pass the image datathrough to a 3D Graphics accelerator 116. Other examples may involvefeature extraction from the image, classification of those features andalerting the main CPU of those results.

[0025] IP 110 could be implemented as a DSP like subsystem with its ownmemory, CPU and I/O capability. A high performance parallel executionunit CPU like the TMS320C80 is preferably used to execute algorithmswhich may employ fast Fourier Transform (FFT) like calculations. Anexample of IP 110 is Ariel Corp's TMS320C80 based Griffin PCI bus imageprocessing board.

[0026] Zoom Video is preferably used to pass the image data in real timeto IP 110 and deliver the results to 3D graphics processor 116. PCI bus108 provides an alternate path for the result or input data. Main CPUsystem 114 may utilize IP 110 as a parallel processor (to itself),pre-processor, or post-processor of image information.

[0027] 3D graphics processor 116 is used to off load time-consuminggraphics operations from the CPU. Although all functions could beimplemented in the main CPU, that would consume a substantial amount ofthe power of present generation of CPUs, leaving little power for othertasks. Graphics processor 116 receives image information from Zoom Videobus 106. It contains a PCI interface, which provides a high bandwidthbus to the CPU for image rendering primitives, background information,text etc. Graphics processor 116 provides near real time rendering ofthe augmentation objects using dedicated hardware specifically designedfor 3D graphics operations.

[0028] Alternately, it is possible for video data to flow at full speed,about 27 Mbytes per second, from the FireWire interface over the PCI busand directly to the memory of graphic processor 116.

[0029] The image generated by graphics processor 116 is sent to adisplay 118. It converts RGB encoded digital data into light. Thepreferred display is small size, low power consumption, and highresolution. Examples of suitable displays are active matrix color liquidcrystal display (LCD), LCD projection panel, digital micromirror device(DMD), and vacuum fluorescent display (VFD).

[0030] CPU system 114 could be a single CPU. Alternatively, it could bea multiprocessing system. Examples are MIPS 10000, DEC ALPHA, SUN UltraSPARC. The preferred system is a Pentium Pro single or multiprocessorsystem (this choice is based on costs and availability of developmenttools).

[0031] The preferred CPU system typically requires a core logic chipset(shown in FIG. 2 as reference numeral 120). It provides the interfacebetween CPU system 114, a main memory 122, and PCI data bus 108.Examples of chipsets are Intel's Orion Core Logic chipset, 440FX, 450GX,and 450KX. The Orion chipset provides multi-processing support for up tofour processors. A PCI bus and an Industry Standard Architecture (ISA)bus (shown as reference numeral 126) are supported. The preferred CPUsystem also requires random access memory (RAM) 122 to provide storagefor program execution.

[0032] It should be noted that the use of 3D graphics processor 116 isoptional. Some microprocessors contain multimedia instructions whichallow multimedia tasks to be easily performed. Examples of suchmicroprocessors are Intel's Pentium-MMX (code named the P55C) and Sun'sUltraSPARC. It should also be pointed out that graphics processors couldbe used in combination with this kind of microprocessors (i.e., havinginstructions designed to execute multimedia instructions) in order toobtain enhanced performance.

[0033] System 100 comprises a mass storage unit 130, which is coupled toPCI bus 108 by a hard disk interface 131. Examples of interface 131 arean Enhanced Integrated Drive Electronics (EIDE) interface and a SmallComputer System Interface (SCSI). Unit 130 provides storage space forGeographic Information Systems (GIS) database information, algorithm andprogram information for IP 110, and operating system software for mainCPU system 114. System 100 contains software (which could be stored inmass storage unit 130 and loaded into RAM 122 or burnt into read-onlymemory) for searching and retrieving data from the GIS database. Thesearching preferably uses position, attitude, and other data foridentifying the location and point of view of camera 102.

[0034] Although there are mass storage units having several gigabytes ofstorage, their physical sizes are too large for the present embodiment.The preferred mass storage unit 130 is a balance between size, weight,cost and performance. At the present time, a rotating magnetic mediumstorage using 1.2 gigabyte 2.5″ technology is considered the preferredstorage unit.

[0035] System 100 also comprises a real time clock (RTC) 134. Itprovides local timekeeping and (optionally) non-volatile storage ofsystem parameters during power off conditions. The design of RTC 134depends on the requirements of CPU system 114 and core chipset 120. Manycompanies, such as Dallas Semiconductors, Inc., BenchmarqSemiconductors, and Chips & Technologies, Inc. manufacture RTCs forvarious CPU system architectures.

[0036] The connection to various peripheral devices is now described.Serial ports (shown as numerals 136, 137 and 160), such as RS232,NMEA-183, and RS422, could be used to provide the connection. The serialports provide serial to parallel conversion, which convertsasynchronously formatted data (received from the peripheral devices) toparallel data. The data is sent over ISA bus 126 to CPU system 114 forprocessing. It should be noted that the present invention is not limitedto using asynchronous serial ports as means for interfacing withperipheral devices. For example, parallel ports or synchronous serialports could also be used.

[0037] An example of a peripheral device that can be connected to serialports 137 is a Global Positioning System (GPS) 140. It derives3-dimensional position information from a GPS satellite navigationsystem (not shown). Typically, the 3-dimensional position information isderived by a “GPS Core” module which measures transit times of theL-band signals broadcast by the twenty four satellites of the GPSconstellation. In the present embodiment, GPS 140 is interfaced via oneof the serial ports using NMEA 183 format data. As an alternative, aproprietary format may be used from one of the many GPS core modulemakers (e.g., Motorola, Garmin, Trimble, Furuno, Ashtech, Rockwell,Plessy, Canadian Marconi, etc.).

[0038] If it is desirable to improve the accuracy of GPS 140, adifferential GPS (DGPS) 141 could be used. It provides correctioninformation to GPS receiver 140 in order to increase the accuracy andremove the effects of selective availability. DGPS is developed by aprecisely surveyed reference GPS receiver base station. Correction dataderived for each satellite is formatted in RCTM-104 format and broadcastvia a communications system to the user. These corrections are appliedto each of the measurements made in the users GPS receiver so as toproduce a more accurate result. DGPS 141 it is interfaced to the GPSreceiver via a serial interface supporting RCTM-104 format data.

[0039] Alternatively, devices that can receive GPS and/or Glonass(Global Navigational Satellite System) signals can be connected toserial port 137. An example of a device that can receive both GPS andGlonass signals is GG24 developed by Ashtech Inc. A further alternativeembodiment is to use real time kinematic surveying techniques. Thesetechniques are able to achieve higher accuracy than DGPS.

[0040] Another peripheral device that may be used in electro-opticsystem 100 is a spread spectrum (SS) radio 144. It provides wirelesscommunication between peer units or between master and slave units.Major advantages of SS are spectrum re-use, simultaneous existence ofmultiple networks, data security and low probability of intercept. Anexample of a SS radio is Proxim RangeLan 2, which operates at 2.4 GHzand has a data rate of 1.6 Mbits per second.

[0041] An accelerometer 148 can also be coupled to one of the serialports. In one embodiment of accelerometer 148, an integrated circuit(e.g., Analog Devices' ADXL05) is used to generate an analog voltagewhich is proportional to the instantaneous acceleration along aspecified axis. The analog voltage can be converted to a digital value(by an analog-to-digital converter) and then serialized so as to becompatible with a chosen serial port communication protocol. Theacceleration information could be used in image stabilization effortsand in augmenting the information from the GPS and tri-axialmagnetometers.

[0042] A tri-axial magnetometer, such as a TCM-2 module from PrecisionNavigation Inc., can also be connected to one of the serial ports. Sucha device provides attitude information, in all three degrees of freedom,regarding the pointing direction of the optical axis of the camera.

[0043] In FIG. 2, a laser range finder 162 is connected to serial port160. An example of a range finder is Leica's Data Disto RS232.

[0044] System 100 also allows various user interface devices 154 to beconnected to ISA bus 126. These interface devices include devices thatcan accept input signal and generate output signals. Examples of userinterface devices are control buttons, switches, optical indicators(e.g., LEDs) and alarms.

[0045] In the present specification, four exemplary applications ofsystem 100 are described. The first application is an electronicbinoculars having a “text box” superimposed on a real image. The textbox contains text data related to the real image. The second applicationis “0-0” visibility navigation system which can help a user to navigatea movable object (e.g., ships, planes, and vehicles) under adversevisual environment. The third application is an object identificationsystem which attempts to identify an object under adverse viewingconditions. The fourth application is an advanced image augmentationapplication which can process, enhance, and augment images.

[0046] Electronic Binocular

[0047] Camera 102 is used to capture a view and delivers a correspondingvideo data to FireWire chipset 104. The data flows to IP 110. In thisapplication, IP 110 is configured as a pass-through device and justpasses the data, without processing, on to graphics processor 116. CPUsystem 114 is not used to process the image data, so none flows over PCIbus 108. CPU system 114 sends text data to graphics processor 116 whichrenders it at points in the image that correspond to the attitude andlocation “text boxes.” Location data is read from GPS 140 and ifappropriate, DGPS 141, via serial port 137. Attitude information is readfrom tri-axial magnetometer 150 via serial port interface 136. GIS datais retrieved from the database on storage system unit 130.

[0048] In one embodiment, CPU system 114 sets the optical and electroniczoom factors in camera 102. Zoom factors are read back along withexposure and iris information from the camera via the FireWire.Information on zoom factors is used by CPU system 114 to properlygenerate augmentation images.

[0049] “0-0 ” Visibility Navigation

[0050] Camera 102 is used to capture a view and deliver a correspondingvideo data to FireWire chipset 104. The data flows to IP 110, whichattempts to extract features from the data using a plurality of frames.Location data is read from GPS 140 and if appropriate, DGPS 141, viaserial port 137. Attitude information is read from tri-axialmagnetometer 150 via serial port interface 136. CPU system 114 retrievesGIS data from storage unit 130 relative to the current position. It thensends “wire frame” graphics to graphics processor 116, which renders andtextures those wire frames into realistic looking images of what“should” be in the field of view of the system.

[0051] Vessel Identification

[0052] Camera 102 is used to capture a view and delivers a correspondingvideo data to FireWire chipset 104. The data flows to IP 110. CPU system114 has previously retrieved (at the user's request) several 3D modelsof ships that are due to pass by. The 3D models of ships are retrievedfrom storage unit 130. This information is sent to IP 110 via PCI bus108. IP 110 searches the field of view (obtained from camera 102) forobjects. Upon identifying an object, it is compared with various aspectsof the 3D models. Upon finding a “match,” information is sent to CPUsystem 114, which sends a signal to alert the user.

[0053] Advanced Image Augmentation

[0054] Camera 102 is used to capture a view and deliver a correspondingvideo data to FireWire chipset 104. The data flows to IP 110, whichenhances the image and attempts to extract features from the data usinga plurality of frames. CPU system 114 has previously loaded IP 110 withthe appropriate algorithm and program information. The enhanced imageflows to graphics processor 116 (via Zoom Video 106) and the features toCPU system 114 (via PCI bus 108). Meanwhile the image data also flows toCPU system 114 (via PCI bus 108) which uses the feature locations toidentify objects in the image and create a “mask” to be used by graphicsprocessor 116 to “remove” those features from the final view. CPU system114 sends the “masks” to graphics processor 116 along with text anddrawing primitives. Graphics processor 116 renders the text, removes the“masked” objects, and renders non-existent objects as directed by CPUsystem 114.

[0055] Location data is read from GPS 140 and if appropriate, DGPS 141,via serial port 137. Attitude information is read from tri-axialmagnetometer 150 via serial port interface 136. CPU system 114 retrievesGIS data relative to the current position from storage unit 130.

[0056] In this application, camera 102 is configured to achieve adesired stabilization factors. Graphics processor 116 is configured todisplay image data (received from Zoom Video bus 106) while applying the“mask”, text and graphics as overlays. Zoom factors are read back alongwith exposure and iris information from camera 104 via FireWire chipset104.

[0057] The invention now being fully described, it will be apparent toone of ordinary skill in the art that any changes and modifications canbe made thereto without departing from the spirit or scope of theinvention as set forth herein. It should be noted that computingtechnology is constantly being developed. New developments can beappropriately used to improve the system disclosed herein. For example,a new peripheral bus called the Universal Serial Bus (USB) may beadvantageously used to connect a large number of peripherals to thesystem of the present invention. Similarly, various solid state memorydevices such as synchronous DRAM, EDRAM, etc. can also be used. Cachememory can be attached to the present system to improve the performance.Accordingly, the present invention is to be limited solely by the scopeof the appended claims.

1. An image processing system comprising: a data processor; animage-delivery mechanism coupled to the data processor; an informationdelivery mechanism coupled to the processor, wherein the informationdelivery mechanism delivers real scene information to the dataprocessor; a graphic processor coupled to the data processor.